SPIN KOSTKA POLYNOMIALS 



JINKUI WAN AND WEIQIANG WANG 

Abstract. We introduce a spin analogue of Kostka polynomials and show that these 
polynomials enjoy favorable properties parallel to the Kostka polynomials. Further 
connections of spin Kostka polynomials with representation theory are established. 



1. Introduction 

1.1. The Kostka numbers and Kostka(-Foulkes) polynomials are ubiquitous in alge- 
braic combinatorics, geometry, and representation theory. A most interesting property of 
Kostka polynomials is that they have non-negative integer coefficients (due to Lascoux 
and Schiitzenberger |LSj ). and this has been derived by Garsia and Procesi |GPj from 
Springer theory of Weyl group representations |Sp| . Kostka polynomials also coincide 
with Lusztig's ^-weight multiplicity in finite dimensional irreducible representations of 
the general linear Lie algebra \Lu\ IKa| . R. Brylinski [Brj introduced a Brylinski-Kostant 
filtration on weight spaces of finite dimensional irreducible representations and proved 
that Lusztig's g-weight multiplicities (and hence Kostka polynomials) are precisely the 
polynomials associated to such a filtration. For more on Kostka polynomials, we refer to 
Macdonald [Mac] or the survey paper of Desarmenien, Leclerc and Thibon |DLT| . 

The classical theory of representations and characters of symmetric groups admits a 
remarkable spin generalization due to Schur [Schj . Many important constructions for 
symmetric groups and symmetric functions admit highly nontrivial spin counterparts, 
including Schur Q-functions and shifted tableaux (cf. e.g. [Mac] ) and Robinson-Shensted- 
Knuth correspondence (see Sagan |Saj). 

The goal of this paper is to add several items to the list of spin counterparts of classi- 
cal theory. We introduce a notion of spin Kostka polynomials, and establish their main 
properties including the integrality and positivity as well as representation theoretic in- 
terpretations. We also introduce a notion of spin Hall-Littlewood polynomials. Our 
constructions afford natural q, t- generalizations in connection with Macdonald polyno- 
mials. The definitions made in this paper look very classical, and we are led to them 
from representation theoretic considerations. Once things are set up right, the proofs of 
the main results, which are based on the classical deep work on Kostka polynomials, are 
remarkably easy. 

There has been a very interesting work of Tudose and Zabrocki |TZ] who defined a 
version of spin Kostka polynomials and spin Hall-Littlewood polynomials (in different 
terminology), adapting the vertex operator technique developed by Jing [Ji] and others 
(cf. Shimozono and Zabrocki |SZj ). Their definitions do not coincide with ours as shown 
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by examples, and the precise connection between the two (if it exists) remains unclear. 
The Q-Kostka polynomials of Tudose and Zabrocki conjecturally admit integrality and 
positivity, but their approach does not seem to easily exhibit the connections to repre- 
sentation theory or afford q, t-generalization as developed in this paper. Throughout the 
paper we work with the complex field C as the ground field. 

1.2. Denote by IP the set of partitions and by IP n the set of partitions of n. Denote by 
SIP the set of strict partitions and by SIP n the set of strict partitions of n. Let A denote 
the ring of symmetric functions in x = (x\,X2, • • •), and let V be the subring of A with a 
Z-basis given by the Schur Q-functions Q^(x) indexed by £ G SIP, cf. [Mac] . 

As an element in the ring A, the Schur Q-functions Q$(x) can be expressed as a linear 
combination in the basis of the Hall-Littlewood functions P fJi (x;t) and we define the spin 
Kostka polynomial K7 (t), for £ G SIP and /i S IP, to be the corresponding coefficient; see 
(12. 7p . Recall that the entries K\^{t), A, [i 6 IP of the transition matrix from the Schur basis 
{s\} to the Hall-Littlewood basis {P^(x;t)} for Z[t] ®% A are the Kostka polynomials. 

Our first result concerns some remarkable properties satisfied by the spin Kostka poly- 
nomials (compare with Theorem 12. 1| where some well-known properties of the usual 
Kostka polynomials are listed). For a partition A G IP with length ^(A), we set 

-w -»«-«*. *<*)-{!: V^zi: 

For £ G SIP, we denote by £* the shifted diagram of £, by Cjj the content, by h*j the shifted 
hook length of the cell (i,j) G £*. Also let be the number of marked shifted tableaux 
of shape £ and weight fx; see Section [2T21 for precise definitions. 

Theorem A. The spin Kostka polynomials (t) for £ G SIP n , /i G IP n have the following 
properties: 

(1) K£ (t) = unless £ > M! K#(t) = 2^>. 

(2) The degree of the polynomial KJ^it) is n(fj,) — 

(3) K^(t) is a polynomial with non-negative integer coefficients. 

(4) AT^l) = K^; K-^(-l) = 2^ . 

(5) ^(t) = ^)n£ ) (i+t i - j ). 

(6) ^ w = iwi^ • 

1.3. It is known (cf. Kleshchev |Kle] ) that the spin representation theory of the symmet- 
ric group is equivalent to its counterpart for Hecke-Clifford algebra % n := C n x CS n , and 
the irreducible !K n -(super)modules are parameterized by strict partitions £ G SIP n . 
The Hecke-Clifford algebra "K n as well as its modules in this paper admit a Z2-graded 
(i.e, super) structure even though we will avoid using the terminology of supermodules. 

For a partition fx G IP n , let IB^ be the variety of flags preserved by a nilpotent matrix of 
Jordan block form of shape fj,, which is a closed subvariety of the flag variety "B of GL n (C). 
The cohomology group H'{ r S> lJ ) of 23 M is naturally an ^-module, and the induced "K n - 
module ind^^H' (H ^) = C n (g> H' (23^) is Z + -graded, with the grading inherited from the 
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one on H'(T>^). Define a polynomial (t) (as a graded multiplicity) by 
(1.1) C"(t) := ^^(dimHomjfjD^en® ^(S^))), 



i>0 

which should be morally viewed as a version of Springer theory (undeveloped yet) of the 
queer Lie supergroups. 

The queer Lie superalgebra q(n) contains the general linear Lie algebra gl(n) as its 
even subalgebra, and its irreducible polynomial representations V(£) are parameterized 
by highest weights £ S SIP with £(£) < n. Let e be a regular nilpotent element in gl(ra), 
regarded as an element in the even subalgebra of q(n). For /i £ f with £(//) < n, using 
the action of e we introduce a Brylinski-Kostant filtration on the weight space V(£)^ and 
denote by 7^(*) the associated polynomial (or q- weight multiplicity). 

The spin Kostka polynomial K7 (t) can be interpreted in terms of graded multiplicity 
C7 u (t) as well as the q- weight multiplicity l7u$) as follows (also see Proposition 13.31 for 
another expression of q- weight multiplicity). 

Theorem B. Suppose £ £ S^n,/^ G Then we have 

(1) K-^t) = 2 M ^ m C^)t^. 

(2) ^ (t) = 2^ X1 7f M (*)• 

Theorem A(6) and Theorem B(l) for = (l n ) (note that "B(in\ = 23) are reinter- 
pretation of a main result of our previous work [WW1| on the spin coinvariant algebra. 
Actually, this has been our original motivation of introducing spin Kostka polynomials 
and finding representation theoretic interpretations. The two interpretations of the spin 
Kostka polynomials in Theorem B are connected to each other via Schur-Sergeev duality 
between q(n) and the Hecke-Clifford algebra [Se] . 

1.4. In Section^ we construct a map $ and a commutative diagram: 

R R- 



ch 



ch" 



A r Q 

where ip given in (|4.ip is as in |Macl III, §8, Example 10], and ch and ch - are characteristic 
maps from the module categories of S n and "K n respectively. The commutative diagram 
serves as a bridge of various old and new constructions, and the use of Hecke-Clifford alge- 
bra provides simple representation theoretic interpretations of some symmetric function 
results in [St] and [Mac] . We further define a spin analogue H~{x;t) of the normal- 
ized Hall-Littlewood function H^(x;t) via the spin Kostka polynomials. We show that 
H~(x\ t) coincides with the image of H^x; t) under the map (p, and it satisfies additional 
favorable properties (see Theorem 14. 4p . 

We also sketch a similar construction of the spin Macdonald polynomials H~(x;q,t) 
and the spin g,t-Kostka polynomials KT(q,t). The use of $ and ip makes such a q,t- 
generalization possible. 
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1.5. The paper is organized as follows. In Section [21 we review some basics on Kostka 
polynomials, introduce the spin Kostka polynomials, and then prove Theorem A. The 
representation theoretic interpretations of spin Kostka polynomials are presented and 
Theorem B is proved in Section [3l In Section [U we introduce the spin Hall-Littlewood 
functions, spin Macdonald polynomials and spin q, i-Kostka polynomials. We end the 
paper with a list of open problems. 

Acknowledgments. We thank Naihuan Jing, Bruce Sagan, and Mark Shimozono for 
helpful discussions and their interests in this work. The first author is partially supported 
by Excellent young scholars Research Fund of Beijing Institute of Technology. The re- 
search of the second author is partially supported by NSF grant DMS-0800280. This paper 
is partially written up during our visit to Academia Sinica and NCTS (South) in Taiwan, 
from which we gratefully acknowledge the support and excellent working environment. 

2. Spin Kostka polynomials 

In this section, we shall first review the basics for Kostka polynomials. Then, we intro- 
duce the spin Kostka polynomials and prove that these polynomials satisfy the properties 
listed in Theorem A. 

2.1. Basics on Kostka polynomials. A partition A will be identified with its Young 
diagram, that is, A = 6 Z 2 | 1 < i < ^(A), 1 < j < Aj}. To each cell (i,j) G A, 

we associate its content Cij = j — i and hook length hij = Aj + A'- — i — j + 1, where 
A' = (A'^A^,...) is the conjugate partition of A. For £ ?, let be the Kostka 
number which counts the number of semistandard tableaux of shape A and weight /x. We 
write |A| = n for A £ CP n . The dominance order on CP is defined by letting 

A > /U -£4> |A| = and Ai + . . . + Aj > [i\ + . . . + fa, Vz > 1. 

Let A, n € CP. The Koksta polynomial K\^{t) is defined by 

(2-1) s x {x) = Y,K x ^t)P^x-t), 

where P^(x;t) and s\(x) are Hall-Littlewood functions and Schur functions respectively 
(cf. [Mac! Ill, §2]). The following is a summary of a long development by many authors. 

Theorem 2.1. (cf. \M.ac\ III, §6]j Suppose A 5/ u € CP n . Then the Kostka polynomial 
Kxfj,(t) satisfies the following properties: 

(1) K Xt t(t) = unless A > fi; K X \(t) = 1. 

(2) The degree of K\^(t) is n(fi) — n(X). 

(3) Kxfj,(t) is a polynomial with non-negative integer coefficients. 

(4) ^(1) = K\fj,. 

(5) K (n)lt (t) = t»M. 

(6) K x(ln) = -j— . 

Let 23 be the flag variety for the general linear group GL n (C). For a partition \x of n, let 
CB M denote the subvariety of CB consisting of flags preserved by the Jordan canonical form 
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of shape [i. It is known |Sp| that the cohomology group H'^B^) of 23 ^ with complex 
coefficient affords a graded representation of the symmetric group S n . Define C X/1 (t) by 

(2-2) C Xfl (t) = £V Hom 5n (5 A ,^ 2l (S M )), 

where S" A denotes the Specht module over S n . 

Theorem 2.2. (cf. [Macl III, §7, Example 8], pB (5.7)]; TTie following holds for 

K Xll {t) = c^(r 1 )t n ^. 

It is well known that the cohomology ring i7*(23) of the flag variety ¥> coincides with 
the coinvariant algebra of the symmetric group S n . Garsia and Procesi |GPj gave a 
purely algebraic construction of the graded 5 n -module i?*(S M ) in terms of quotients of 
the coinvariant algebra of symmetric groups as well as a proof of Theorem 12.21 

Denote by {ei, . . . , e n } the basis dual to the standard basis {En | 1 < i < n} in the 
standard Cartan subalgebra of gl(n), where En denotes the matrix whose (i, i)th entry is 
1 and zero elsewhere. Let L(X) be the irreducible gl(ra)-module with highest weight A for 
A G ? with £(X) < n. For each \x G 7 with £{[i) < n, define the g-weight multiplicity of 
weight /i in L(X) to be 

where the product n o >o ^ s over au positive roots {e^ — ej | 1 < i < j ' < n} for 
gl(n) and [e^]/(e £1 , . . . , e e ") denotes the coefficient of the monomial in a formal se- 
ries f(e ei , . . . ,e 6n ). According to a conjecture of Lusztig proved by Kato |Ka[ ILuj . we 
have 

(2-3) K^(t) = roj(t). 

Let e be a regular nilpotent element in the Lie algebra Ql(n). For each /i G 7 with 
£(li) < n, define the Brylinski-Kostant filtration on the weight space L(X) fJi by 

OCJ^jCjl^C... 

with 

J e fc (L(A) At ) = {v G L(X), | e k+1 v = 0}, 
where we assume J~ 1 (L(A) /i ) = {0}. Define a polynomial 7a^(£) by 
7v (i) = £ (dim J e fc (L(A) M )/Jr i (L(A) M ))t fc . 

fc>0 

The following theorem is due to R. Brylinski (see |Br( Theorem 3.4] and (|2.3p ). 
Theorem 2.3. Suppose X,fJ, G 7 ^(A) < n and £(fi) < n. Then we have 

K Xfl (t) = 1Xfl (t). 
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2.2. Schur Q-functions and spin Kostka polynomials. Given a partition A 6 IP, 
suppose that the main diagonal of the Young diagram A contains r cells. Let a% = Ai — i 
be the number of cells in the ith row of A strictly to the right of (i, i), and let /3j = A^ — i 
be the number of cells in the ith column of A strictly below (i,i), for 1 < i < r. We 
have ai > atz > ■ ■ ■ > a r > and j3\ > 02 > ■ ■ ■ > f3 r > 0. Then the Frobenius 
notation for a partition is A = (a±, . . . , a r \/3i, . . . , /3 r ). For example, if A = (5,4,3,1), 
then a = (4, 2, 0), /3 = (3, 1,0) and hence A = (4, 2, 0|3, 1, 0) in Frobenius notation. 

For a strict partition £ £ STVn let be the associated shifted Young diagram, that is, 

e = {(i,j) I 1 <J <& + *-!} 

which is obtained from the ordinary Young diagram by shifting the /cth row to the right 
by k — 1 squares, for each fc. Given £ € §y n with £(£) = £, define its double partition (or 
double diagram) £ to be £ = (£i, . . . , — 1, ^2 — 1 ; • • • , ^ — 1) in Frobenius notation. 
Clearly, the shifted Young diagram £* coincides with the part of £ that lies above the 
main diagonal. For each cell G £*, denote by h*j the associated hook length in the 

Young diagram £, and set the content Cij = j — i. 

For example, let £ = (4, 2, 1). The corresponding shifted diagram and double diagram 
are 



The contents of £ are listed in the corresponding cells of as follows: 








2 


CO 







1 














The shifted hook lengths for each cell in are defined as the usual hook lengths for the 
corresponding cell in the double diagram £, as follows: 
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Denote by P' the ordered alphabet {l'<l<2 / <2<3'<3---}. The symbols 
1',2',3'j . • • are sa id to be marked, and we shall denote by \a\ the unmarked version of 
any a 6 P'; that is, \k'\ = \k\ = k for each k £ N. For a strict partition £, a marked 
shifted tableau T of shape £, or a marked shifted ^-tableau T, is an assignment T : £* — > P' 
satisfying: 

(Ml) The letters are weakly increasing along each row and column. 
(M2) The letters {1, 2, 3, . . .} are strictly increasing along each column. 
(M3) The letters {1', 2', 3', . • •} are strictly increasing along each row. 

For a marked shifted tableau T of shape £, let be the number of cells (i,j) £ £* 
such that \T(i,j)\ = k for k > 1. The sequence (ai,a2>ci3> . . .) is called the weight of T. 
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The Schur Q- function associated to £ can be interpreted as (see [Sa l ISt j iMacj ) 

T 

where the summation is taken over all marked shifted tableaux of shape £, and x T = 
x^x^Xg 3 -- - if T has weight (01,02,03,...)- Set 

= #{T I T is a marked shifted tableau of shape £ and weight /x}. 

Then we have 

(2.4) Q^x) = J2K^(x). 

It will be convenient to introduce another family of symmetric functions q\(x) for 
any partition A = (Ai,A2,...) as follows: qo(x) = 1, q r (x) = Q^(x) for r > 1, and 
q\(x) = qx 1 (x)qx 2 (x) ■ ■ ■ . The generating function Q(u) for q r (x) is 

(2-5) J2lr(x)u r = Q(u) = H\±^. 

^-^ 1A - 1 — XiU 

r>0 i 

We will write q r = q r (x), etc., whenever there is no need to specify the variables. Let 
T be the Z-algebra generated by q r ,r > 1, that is, 

(2.6) r = z[ 9l , ©,...]. 

It is known that the set {Q^ | £ G S3 3 } forms a Z-basis of T. 

Definition 2.4. The spin Kostka polynomials K^^t) for £ € S3 3 and [i G IP are given by 

(2.7) Q s (x) = ^^-(t)P M (x;t). 

2.3. Properties of spin Kostka polynomials. For £ G SIP, write 

(2.8) Qs(x) = J^ A s A (x), 

for some suitable constants 6^ A . 

Proposition 2.5. T/ie following holds for £ G SIP and fi G IP: 

Proof. By (|2.ip and (|2.8p . one can deduce that 

(t)P M (x;t) = ^6 5A K AM (t)P M (x;t). 

The proposition now follows from the fact that the Hall-Littlewood functions P^(x;t) are 
linearly independent in ®% A. □ 
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The usual Kostka polynomial satisfies that K\^(0) = Sx^. It follows from Proposi- 
tion E2D that 

For £ G §9, A G 5 , set 

(2.9) g 5A = 2-^)6^. 

Lemma 2.6. [St, Theorem 9.3] [Macl III, (8.17)] T/ie following holds for £ G S3 3 , A G IP: 

(2.10) € Z + ; ^ A = unless £ > A; = 1. 

Stembridge [St] proved Lemma 12.61 by giving a combinatorial formula for i n terms 
of marked shifted tableaux. We shall give a simple representation theoretic proof of 
Lemma 12.61 in Section [3.41 for the sake of completeness. 

Proof of Theorem A. Combining Theorem !2.1l fl)-(3). Lemma 12.61 and Proposition 12.51 
we can easily verify that the spin Kostka polynomial (t) must satisfy the properties 
(l)-(3) in Theorem A. 

It is known that P^(x;l) = and hence by (|2.4p we have K7(l) = K7 u . Also, 

Q ( = 2t®P{(x;-l), and {P^(x;-1) \ fx G 5 } forms a basis for A (see [Macl p.253]). 
Hence (4) is proved. 

By [Mac] III, §3, Example 1(3)] we have 

(2.H) n = £ tnM IK 1 + W*; *)■ 

i>i m i=i 

Comparing the degree n terms of (|2.11|) and ([2.5p . we obtain that 

Hi*) 

Q {n) {x) = q n {x) = Y, t^Uil+t^P^t). 

li&n j = l 

Hence (5) is proved. 

Part (6) actually follows from Theorem B(l) and the main result of [WWlj . and let us 
postpone its proof after completing the proof of Theorem B(l). □ 

3. Spin Kostka polynomials and representation theory 

In this section, we shall give two interpretations of spin Kostka polynomials in repre- 
sentation theory. 

3.1. The Frobenius characteristic map ch. Denote by S^-mod the category of finite 
dimensional S^-modules. Let R n = K(S n -mod) be Grothendieck group of the category 
S^-mod and set 

R = Q)Rr, 

n>0 

Recall that R n admits an inner product by declaring the irreducible characters to be 
orthonormal. Also there exists an inner product (, ) on the ring A such that the Schur 
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functions s\ form an orthonormal basis. The Frobenius characteristic map ch : R — > A 
preserves the inner products and it satisfies that 

(3.1) ch([S A ]) = s x , 

(3.2) ch(indg«l) = h x , A€lP n , 
where 1 denotes the trivial character. 

3.2. Hecke-Clifford algebra Ji n and the characteristic map ch~. A superalgebra 
A = AqQAj satisfies Ai ■ Aj C A- l+ j for i, j G ^2- Denote by C n the Clifford superalgebra 
generated by the odd elements ci,... , c n , subject to the relations c? = l,(%Cj = —CjCi 
for 1 < i j < n. The symmetric group S n acts as automorphisms on the Clifford 
algebra S n by permuting its generators, and the Hecke-Clifford algebra is defined to be 
the semi-direct product Ji n = C n x CS n with aci = c a ^a, for a G S n , 1 < i < n. Note 
that the algebra !K n is naturally a superalgebra by letting each a € S n be even and each 
Ci be odd. 

A module over a superalgebra, e.g. 5C n , is always understood to be Z2-graded in this 
paper. It is known [Jol[Sel[St] (cf. |Klej ) that there exists an irreducible ?{ n -module D* for 
each strict partition £ G SCP ri and {D^ \ £ G S'J'n} forms a complete set of non-isomorphic 
irreducible !K n -modules. A superalgebra analogue of Schur's Lemma states that the endo- 
morphism algebra of a finite dimensional irreducible module over a superalgebra is either 
one dimensional or two dimensional. It turns out that [Jo} ISe] 

(3.3) dimHom^p 5 ,^) = 2 5 ^ . 

Denote by ^K n -smod the category of finite dimensional IK n -supermodules. Let R~ be 
the Grothendieck group of the category ^{ n -smod and define 



n>0 

Recall the ring V from (|2.6p and set Tq = Q 0z T. As a spin analogue of the Frobenius 
characteristic map ch, there exists an isomorphism of graded vector spaces [Jo] 

<h :R q — ► r Q 

(3.4) [£>*] ^ 2- L V U Q € , 

(3.5) ind^l^v 

It is useful to note that ch~ is related to ch as follows: 

(3.6) ch-(C) = ch(res^ n C), for C E R~. 

3.3. Spin Kostka polynomials and graded multiplicity. Up to some 2-power, g^\ 
has the following representation theoretic interpretation. 

Lemma 3.1. Suppose £ G SCPrn A G lP n . The following holds: 



dimHom^(^,ind^ n S A ) = g^. 



10 WAN AND WANG 

Proof. Since the IK n -module indjj^g S x is semisimple, we have 

dim Homj{ n (D^, indj^ S x ) = dim Hom Mn (indg^S* L> € ) 

= dim Hornet (S A , resg^-D*) 
=( SA ,ch(res^ n ^)) 
=( SA ,ch^( J D«)) 

= (s x ,2 —Q^(x)) 

=2 2 g^x, 

where the second equation uses the Frobenius reciprocity, the third equation uses the fact 
that ch is an isometry, the fourth, fifth and sixth equations follow from (|3.6p . (|3.4p and 
(|2.8p . respectively. □ 



Proof of Theorem B(l). Suppose £ £ §T n and // € IP n ,. By Proposition 12.51 and Theo- 
rem [221 we obtain that 

K Jft) = E hxKx,{t) = E ^ACA^i- 1 )^^. 

On the other hand, recalling the definition of C^{t) from and the definition of C\^{t) 
from (|2.2p , we have by Lemma 13.11 that 



= E f (dimHom Mn (^,ind^^(S 
= £ CV(i) dimHom^C^, ind^5 A ; 



= 2 2— E hxC\»{t). 

Now Theorem B(l) follows by comparing the above two identities. □ 

With Theorem B(l) at hand, we can complete the proof of Theorem A. 

Proof of Theorem A(6). Suppose £ £ §"P n . Observe that "Bnn) = H and it is well known 
that H'CB) is isomorphic to the coinvariant algebra of the symmetric group S n . Hence 
by [WWII Theorem 3.5] and i$EM we obtain that 



2~ 



no-HQ 



^(i-^-i 2 )-(i-^n ( ,, M .(i+i c 



L(ij)ee 

where £* is the shifted Young diagram associated to £ and Cjj , h*j are contents and shifted 
hook lengths for a cell € £, respectively. This together with Theorem B(l) gives rise 
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to 



_ t -l )(1 _ r 2) . . . (1 _ r ») n ( „, e£ ,(l + t 



r n-n(0+E (l , 3)ee Kj (1 _ _ j2) . . . (1 _ r) n (ij . )ee (1 + ) 

t n(0(l _ t){ l _ g) , , , (1 _ n n (tj)ge (l + t«i) 

where the last equality can be derived by noting that the contents are 0, 1, — 1 

and the fact (cf. [Mac I III, §8, Example 12]) that in the ith row of £*, the hook lengths 
h*j for z < j < + i — l are 1,2,..., & + & + 6+2, ■ ■ • , & + 6 wit h exception 
£i - - 6+2, — □ 

3.4. Spin Kostka polynomials and g-weight multiplicity. The queer Lie superalge- 
bra, denoted by q(n), can be viewed as the subalgebra of the general linear Lie superal- 
gebra 0[(n|n) consisting of matrices of the form 

'a 6 N 
b 



(3.7) 



where a and 6 are arbitrary nxn matrices. Let q = q(n) and I(n\n) = {I, . . . , n, 1, . . . , n}. 
The even (respectively, odd) part Qq (respectively, 0j) consists of those matrices of the 
form (|3.7p with b = (respectively, a = 0). Denote by £7jj for i,j € 7(n|n) the standard 
elementary matrix with the (i,j)ih entry being 1 and zero elsewhere. Fix the triangular 
decomposition 

= rT © f) 0n + , 

where h, (respectively, n + , n~) is the subalgebra of q which consists of matrices of the 
form (|3.7p with a, b being arbitrary diagonal (respectively, upper triangular, lower trian- 
gluar) matrices. Let b = f) © n + . 

Let {ej | i = 1, . . . , n} be the basis dual to the standard basis {En + Ej j \ i = 1, . . . , n} 
for the even subalgebra f)Q of f), where fjg is identified with the standard Cartan subalgebra 
of gl(n) via the natural isomorphism cj(n)g — Qi(n). With respect to t)g we have the root 
space decomposition = (j ffl QgA g Q with roots A = {ej — ej\l < i ^ j < n}. The set 
of positive roots corresponding to the Borel subalgebra b is 

A + = {ej - €j\l < i < j < n}. 

Noting that [fjj, fjj] = f)g, the Lie superalgebra h, is not abelian. For A € X^=i c ^0' 
define the symmetric bilinear form (•, -)\ on fjj by (v,w}\ := A([-y,w]). Let f)j C b,j be a 
maximal isotropic subspace and consider the subalgebra h/ = h,g©f)j. The one-dimensional 
fjg-module C^a, defined by hv\ = \(h)v\, extends trivially to b/. The induced f}-module 
W\ := Ind[j,Cf a is irreducible. Extend W\ to representation of b by letting n + WA = 0. 
The induced g-module ind^WA has a unique irreducible quotient, denoted by V(X). We 
have a weight space decomposition V(X) = ® V(X)^, where a weight /i can be identified 
with a composition (fi\, . . . , fi n ). 
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For £ € §y with < n, the q(ra)-module V(£) is finite dimensional. Moreover, 
according to Sergeev [Se], the character of V(£) (by setting Xi = e ei ) is 

(3.8) chy(0 = 2 2Qt(xi,...,Xn). 

Regarding a regular nilpotent element e in g[(n) as an even element in q(n), we define a 
Brylinski-Kostant filtration on the weight space V(£)p by 

where 

:= 6 I e k+1 v = 0}. 

Define a polynomial 7^(i) by 

T&W = E (dimjJ^COMJ/J*- 1 ^^))**- 

fc>0 

Recall that L(X) denotes the irreducible representation of gl(n) with highest weight A 
and recall from (|2.9j) . Up to the same 2-power as in Lemma 13.11 has the following 
interpretation of branching coefficient. 

Lemma 3.2. As a g\(n) -module, V(£) can 6e decomposed as 

V(0 = © 2 Ji V £1 ^ A L(A). 

AeIP^(A)<n 

Proof. It suffices to verify on the character level. The corresponding character identity 
indeed follows from (j2.8|) . (|2.9p and (|3.8|) . as the character of L(X) is given by the Schur 
function s\. □ 

Now we give a proof of Lemma [2.61 based on representation theory of q(n) as promised. 
It is also possible to give another proof based on representation theory of Hecke-Clifford 
algebra "K n . 

Proof of Lemma \2. 61 It follows by Lemma [3.21 that > 0, and moreover, g^\ = unless 
£ > A (the dominance order for compositions coincide with the dominance order of weights 
for q(n)). The highest weight space for the q(n)-module V(£) is which has dimension 

2 S a (fl . Hence, g^ = 1, by Lemma 13.21 again. 

By (UTS]), 2~ Mt ¥ m diV(0 = 2~ i ®Q(x 1 ,...,x n ), which is known to lie in A, cf. 
[Mac] (this fact can also be seen directly using representation theory of q(n)). Hence, 
2~^'Q(x\, . . . ,x n ) is a Z-linear combination of Schur polynomials s\. Combining with 
Lemma l3.2| this proves that g%\ € Z. □ 

We are ready to establish the Lie theoretic interpretation of spin Kostka polynomials. 

Proof of Theorem B(2). The Brylinski-Kostant filtration is defined via a regular nilpotent 
element in gl(n) = q(rc.)o> and thus it is compatible with the decomposition in Lemma[ 
Hence, we have 

A 
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It follows by the definitions of the polynomials 7^(£) and 7a^(0 that 

A 

Then by Theorem 12.31 we obtain that 

= E 2 ^^^A^A M (i) = X) 2 * ^A^M*)- 
A A 

This together with Proposition 12.51 proves Theorem B(2). □ 

The interpretation of spin Kostka polynomials as q- weight multiplicity can take another 
form. 

Proposition 3.3. Suppose £ £ SO 5 and p £ J urei/t £(£) < n and < re. Then we have 
K7(t) = 2^[e ^^ 1 - e ~ a \ cW(0- 

[ J n« e A+(i-te- Q ) 

Proof. It follows from (12, 3j) that 

** (,) " [e " ] ffera ch£(A) 

for A G CP with £(X) < n. Hence by Propostion 12.51 and Lemma 1331 one deduces that 

na g A + (l-Q 

Ahn 



= E ^ ■ L a te"4 chL(A) 



loeA4 v ' Ae3V(A)<n 

n« e A + d-te-°) 

□ 

4. Spin Hall-Littlewood and spin Macdonald polynomials 

In this section, we introduce the spin Hall-Littlewood polynomials and establish their 
main properties. We also formulate the q, t-generalizations of spin Kostka polynomials 
and Macdonald polynomials. 

4.1. A commutative diagram. Recall a homomorphism ip [Mac! Ill, §8, Example 10] 
defined by 

if-. A^r, 

2p r , for r odd, 

0, otherwise, 



(4.1) (p( Pr 

Le rth power 



where p r denotes the rth power sum symmetric function. Denote 

Prt r 
r 



n>0 i r>l 



11 



WAN AND WANG 



Pr£_ 

r 



Noting that Q(t) from (|2.5H can be rewritten as 

Q(t) = exp (2 

r>l,r odd 

we obtain that 

(4.2) <p(H(t))=Q(t). 
Hence, we have <p(h n ) = q n for all n, and 

(4.3) (pQiJ = q„, V/i G 7. 
For each n > 0, we define a functor 

$ n : S'n-mod 



J{ n -smod 



by sending M to ind C g M. Such a sequence of functors {& n } induces a Z-linear map on 
the Grothendick group level: 

$ : R — >R-, 

by letting $([M]) = [$ n (M)] for M G 5 n -mod. We shall show that the map $ : R -> R~ 
(or the sequence {^n}) is a categorification of 00 : A — > T. 

Recall that carries a natural Hopf algebra structure with multiplication given by 
induction and comultiplication given by restriction |Zej . In the same fashion, we can 
define a Hopf algebra structure on R~ by induction and restriction. On the other hand, 
Aq = Q[pi,P2)£>3) • • •] is naturally a Hopf algebra, where each p r is a primitive element, 
and r<Q) = Q[pi,P3,P5, • • •] is naturally a Hopf subalgebra of Aq. The characteristic map 
ch : Rq — > Aq is an isomorphism of Hopf algebras (cf. |Zej). A similar argument easily 
leads to the following. 

Lemma 4.1. The map ch~ : Rq — » Tq is an isomorphism of Hopf algebras. 

Proposition 4.2. The map $ : Rq — > Rq is a homomorphism of Hopf algebras. More- 
over, we have the following commutative diagram of Hopf algebras: 



(4.4) 

Proof. Using (j3.2[) and 



R 

ch 

A, 

3D we have 



(,7( 



^(ch(ind^l)) 



On the other hand, it follows by (13. that 



ch-($(ind^l)) = ch-(ind^l) = q». 

This establishes the commutative diagram on the level of linear maps, since R n has a 
basis given by the characters of the permutation modules ind C5 n l for fi G CP n . 
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It can be verified easily that <p : Aq — > Tq is a homomorphism of Hopf algebras. Let 
us check that <p commutes with the comultiplication A. 

A(ip(p r )) = A(2p r ) = 2(p r ® 1 + 1 ® p r ) = (tp ® tp)(A(p r )), for odd r. 
A((p(p r )) = = (if ® p)(A(p r )), for even r. 

Since both ch and ch~ are isomorphisms of Hopf algebras, it follows from the commuta- 
tivity of (|4,4p that $ : Rq — > Rq is a homomorphism of Hopf algebras. □ 

4.2. Spin Hall-Littlewood functions. Denote by H^(x;t) the basis of A dual to the 
Hall-Littlwood functions P^(x;t) with respect to the standard inner product (•,•) on A 
such that Schur functions form an orthonormal basis. It follows by the Cauchy identity 
and (O) that 



(4-5) n = J2 H ^Wy^> 

(4.6) H„(x;t) = J2Kx„(t)sx(x). 

xev 

Recall that in A-ring formalism, the symmetric functions in x(l — t) are defined in terms 
of Pfc((l — t)x) = (1 — t k )pk(x). Actually, the symmetric functions P^{x]t) and H^{x]t) 
are related to each other as (cf. jDLTj ) 

1 

bjty 

where b^(t) = f\i>i T\j?=i\l ~ ^) an d m i(l L ) denotes the number of times i occurs as a 
part of ji. 



(4-7) P^x;t) = -—H^(l-t)x;t), 



Definition 4.3. Define the spin Hall-Littlewood function (x;t) for /i £ CP by 

(4.8) H-(x;t) = ^ 2-*®K^(t)Qz(x). 

For A G CP, let S x G T be the determinant (cf. [Macl III, §8, 7(a)]) 

S x = det(qxi-i+j). 
It follows by the Jacobi-Trudi identity for s\ and (|4.3|) that 

(4.9) ip(sx) = S x . 

Applying p to the Cauchy identity f\i j i-xy = J2xev s \i x ) s \(y) an d using (|4.2p with 
t = m, we obtain that 

(4.10) n^|f = £*(*)-a(»). 

i,j>i iyj AelP 

It follows by the commutative diagram (|QI) and ([49]) that S\{x) = ch(C n ® S x ). This 
recovers and provides a representation theoretic context for [Mac} III, §8, 7(c)], as the 
Clifford algebra C n is isomorphic to the exterior algebra A(C n ) as ,S n -modules. 
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Theorem 4.4. The spin Hall-Littlewood functions H„ (x; t) for fj, 6 "P satisfy the following 
properties: 

(1) ip(H„(x;t)) = H-(x;t). 

(2) H-(x;l)=q fi (x). 

(3) H-(x;0) = S fl (x). 

w I* v ' [0, otherwise . 

(5) H~(x;t) G Z[i] ® z r,Vp € IP; {#f(x;i) | £ € SJ 1 } /orms a basis o/Z[i] ® z T. 
1 + XiVj 



Proof. By ()4.10p and the Cauchy identity for Schur Q-functions, we have 

£ 2^«>Q e (x)Cfc(y) = £S A (x) SA (y). 

Substituting with Q^(y) = Ylxev^^S^^iv) in the above equation, we obtain that 
(4.11) S x (x) = ]T g^xQ^x). 

Part (1) can now be proved using (|4"U|) . ([43]) . (|4.1ip . Proposition [231 and (|4T8|) : 

= £ 9£\ K \iJ,(t)Qd x ) 
geST.Aey 

=H~{x-t). 

Since i^(x;0) = and i^(x; 1) = (2) and (3) follow from l|£5]> . (|Q]> and (1). 
Also, (4) follows by Theorem A(4) and the definition of H~(x;t). 

We have 2~ i( -^K^(t) G Z[i] by Theorem A(3) and Q ? (x) G T, and hence by (JO]) . 
H~(x;t) G Z[i] (8>z r. By (|4.8j) and Theorem A(l)(3), the transition matrix between 
{H^(x;t) | £ G S3 5 } and {(^(x) | £ G SCP} is unital upper triangular with entries in Z[i]. 
Therefore, | £ G S3 3 } forms a basis of Z[t] <£>^ T since so does {Q$(x) | £ G ST}. 

This proves (5). 

(6) follows by applying the map <p to both sides of (j4.5[) in x variables and using (14. 2j) 
and (1). □ 

4.3. Spin Macdonald polynomials and spin q, i-Kostka polynomials. Denote by 
H\(x;q,t) the normalized Macdonald polynomials, which is related to the Macdonald's 
integral form J\(x;q,t) by 

H x (x;q,t) = J(x/(l-t);q,t) 
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in A-ring notation (cf. [Mac! IGH] ) . Inspired by Theorem 14.4( 1). we make the following. 
Definition 4.5. The spin Macdonald polynomials H~(x;q,t) for fi G CP is given by 

H~(x;q,t) = ip(H^(x;q,t)). 
The spin q,t-Kostka polynomials Kg (q,t) for £ € S3 3 and fi G CP are given by 

fl--(*;g,t) = ^ 2-<«)^-(g,t)Q { (x). 

Compare with (|4.6p for Kostka polynomials and (j4.8j) for spin Kostka polynomials. 
The classical <7,i-Kostka polynomial K\n(q,t) can be characterized as follows: 

(4.12) H^x; q,t) = J2 K Xfl (q, t)s x (x). 

According to Garsia and Haiman |GFH IHai] , there is a Z + x Z+-graded regular represen- 
tation of S n , parameterized by \i G CP n : 

»,j>0 

such that 

(4.13) ]T dimHom 5ri (5 A ,^)^f = K^{q,r x )t n ^ . 

i,j>0 

In particular, this established a conjecture of Macdonald [Mac] that K\^(q,t) G Z + [(/,£]. 
For fi G CP n we consider the doubly graded CK n -module 3> n (i? At ) = C n <g> i?^, and set 

: = Z) d™Hom M „( J D^e n ®^)g^ i . 

i,J>0 

Proposition 4.6. XTie following identities hold for £ G SCP and [i G CP: 
(4-14) K^(q,t) = J2kxKx fl (q,t). 

(4.15) C^(9,t) = 2- m ^ m K^(q,t- 1 )t n M. 

Proof. The identity (|4.14p follows by applying the map (p to (|4.12p and using (|4.9p and 

(BHD. 

Suppose £ G SCP n and /i G CP n . We compute by Lemma [34] and (|4.13p that 
= E ^^dimHo mj£n (^,ind^ n ^) 

i,j>0 



Z ^(?,* _1 )* n(M) dimHom^(D^ind^ n 5 ; 



Ae3>« 



AG?„ 

The identity (|4.15p follows from this and (j4.14j) . □ 
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Note that K^(0,t) = K^{t), and C^(l, 1) = 2~' (e V (0 (1, 1) is the degree of L>«. 
We leave it to the reader to formulate further properties of spin Macdonald polynomials 
and spin q, i-Kostka polynomials. 

4.4. Discussions and open questions. Let {H^(x;t) | £ 6 S3 5 } be the basis dual to 
{H~(x; t) | £ G SO 3 } in T with respect to the inner product Q c ) = 2 £ K)(% C for £, £ € S3 5 , 
or equivalently, 

(4.16) » t (.n /)//,(,,:/ 1. 



It follows that 



= ^^-(x;t)^(y(l-t);t). 

However, the spin analogue of the relation (|4.7h . i.e., a similar relation for H^(x;t) and 

its dual basis H^(x; t) does not hold, as can be shown by examples for n = 3. 
In the case £ = (n), we have 

H- n) ((l - t)x;t) = q n ((l-t)x) = (-l) n g n (x;t), 
where g n (x;t) is defined by 

r>0 i 

Curiously the function g n (x;t) also appears in our calculation of characters of Hecke- 
Clifford algebra in |WW2j . 

According to Lascoux and Schiitzenberger |LSj . the Kostka polynomial Kx^t) has an 
interpretation in terms of the charge of semistandard tableaux of shape A and weight fi. 
This naturally leads to the following. 

Question 4.7. Let £ £ SIP, fi £ 7 with |£| = \n\. Find a statistics spin charge on marked 
shifted tableaux, denoted by sch(T), such that (t) = ^ T t sch ( T ) where the summation 
is taken over all marked shifted ^-tableaux T of weight /i. The spin charge is expected 
to be independent of the marks on the diagonal of a shifted tableau to account for the 
factor 2^) for Kr(t). 

A possible approach toward spin charge would be using the quantum affine queer 
algebra introduced by Chen and Guay ICG] . 



Example 4.8. The positive integer polynomials 2 ^^K^ (t) for n = 3, 4 are listed in 
matrix form as follows. 



(n = 3) 



£\/i (3) (2,1) (l 3 ) 
(3) 1 1 + t l + t + t 2 + t 3 
(2,1) 1 t + t 2 
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£V (4) (3,1) (2,2) (2,1,1) (l 4 ) 
(n = 4) (4) 1 1 + t t + t 2 l + t + t 2 + t 3 l + t + t 2 + 2t 3 + t 4 + t 5 + t 6 

(3,1) 1 1 + t l + 2t + t 2 t + 2t 2 + 2t 3 + 2t 4 + t 5 

These examples show that the spin Kostka polynomials given in this paper and those by 
Tudose and Zabrocki |TZ| using vertex operators are not the same. 

Question 4.9. Does there exist a vertex operator interpretation for our version of spin 
Hall-Littlewood polynomials? 

Recall that the Kostka polynomial Kx^t) for A, fi € CP is symmetric in the sense that 

for some m\u G Z. Example 14.81 seems to indicate such a symmetry property for the spin 
Kostka polynomials as well, as Bruce Sagan suggested to us. 

Question 4.10. Does there exist rnT £ Z so that the spin Koskta polynomial (t) for 
£ € S3 3 , /i € CP satisfies 

A7 M (t) = r^- (t- 1 )? 

Finally, the spin g, t-analogue deserves to be further studied. 

Question 4.11. Develop systematically a combinatorial theory for the spin Macdonald 
polynomials and spin q, i-Kostka polynomials. 
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